Skip to content

feat: [6659773700] relax window query SELECT list restrictions#35302

Closed
Simon9997 wants to merge 7 commits into
3.0from
feat/3.0/6659773700
Closed

feat: [6659773700] relax window query SELECT list restrictions#35302
Simon9997 wants to merge 7 commits into
3.0from
feat/3.0/6659773700

Conversation

@Simon9997
Copy link
Copy Markdown
Contributor

Description

Issue(s)

  • Close/close/Fix/fix/Resolve/resolve: Issue Link

Checklist

Please check the items in the checklist if applicable.

  • Is the user manual updated?
  • Are the test cases passed and automated?
  • Is there no significant decrease in test coverage?

@Simon9997 Simon9997 requested a review from guanshengliang as a code owner May 9, 2026 07:38
Copilot AI review requested due to automatic review settings May 9, 2026 07:38
@Simon9997 Simon9997 requested review from a team and dapan1121 as code owners May 9, 2026 07:38
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new window projection mode to the query engine, allowing for more flexible handling of scalar expressions and projections in windowed queries. It adds the necessary infrastructure, including new enum types and AST/planner updates, to support this mode. My review identified a potential logic error where auxiliary columns (pseudo-columns and group keys) required for HAVING or ORDER BY clauses might be missed when the projection mode is active, as well as a concern regarding the robustness of the logic used to identify degenerate projection lists in subqueries.

Comment thread source/libs/planner/src/planLogicCreater.c
Comment thread source/libs/planner/src/planLogicCreater.c Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR expands TDengine’s window-query capabilities by relaxing SELECT-list restrictions for window queries (especially “projection-mode” windows without aggregates), and updates the planner/executor to support the new behavior with additional CI coverage.

Changes:

  • Add projection-mode support for window queries (raw columns / scalar expressions without aggregates) across planner splitting, logic planning, translation checks, and executor indefinite-rows runtime.
  • Introduce SCALAR / AGG window-mode selectors in the SQL grammar and tokenizer.
  • Add a comprehensive new window-projection in/ans test suite and update existing tests that previously expected syntax errors.

Reviewed changes

Copilot reviewed 40 out of 47 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
test/ci/cases.task Add CI entry to run the new window-projection pytest suite.
test/cases/query/test_window_projection.py New pytest suite building datasets and running negative + in/ans validations for window projection.
test/cases/query/in/test_window_projection_small.in New small dataset query set for in/ans comparison (includes many window projection scenarios).
test/cases/query/in/test_window_projection_mode.in New coverage for SCALAR/AGG mode variants in window projection queries.
test/cases/query/in/test_window_projection_large.in New multi-vgroup in/ans workload for window projection queries.
test/cases/query/in/test_window_projection_largedata.in New cross-block (large row count) in/ans workload for window projection.
test/cases/query/in/test_window_projection_partition.in New partition-by focused in/ans workload for window projection.
test/cases/query/in/test_window_projection_edge.in New empty/single-row edge-case in/ans workload.
test/cases/query/ans/test_window_projection_small.ans Expected output for the small in/ans workload.
test/cases/query/ans/test_window_projection_mode.ans Expected output for the mode-variation workload.
test/cases/query/ans/test_window_projection_large.ans Expected output for the large in/ans workload.
test/cases/query/ans/test_window_projection_largedata.ans Expected output for the cross-block workload.
test/cases/query/ans/test_window_projection_partition.ans Expected output for the partition workload.
test/cases/query/ans/test_window_projection_edge.ans Expected output for the edge-case workload.
test/cases/13-TimeSeriesExt/06-SessionWindow/test_session.py Update assertions: some session window queries previously expected to error are now expected to succeed.
test/cases/13-TimeSeriesExt/04-StateWindow/test_state_window.py Update assertion: a state_window query previously expected to error is now expected to succeed.
test/cases/11-Functions/01-Scalar/test_fun_sca_upper.py Remove scalar+interval query from the “should error” list (behavior change).
test/cases/11-Functions/01-Scalar/test_fun_sca_substr.py Remove scalar+interval query from the “should error” list (behavior change).
test/cases/11-Functions/01-Scalar/test_fun_sca_rtrim.py Remove scalar+interval query from the “should error” list (behavior change).
test/cases/11-Functions/01-Scalar/test_fun_sca_ltrim.py Remove scalar+interval query from the “should error” list (behavior change).
test/cases/11-Functions/01-Scalar/test_fun_sca_lower.py Remove scalar+interval query from the “should error” list (behavior change).
test/cases/11-Functions/01-Scalar/test_fun_sca_length.py Remove scalar+interval query from the “should error” list (behavior change).
test/cases/11-Functions/01-Scalar/test_fun_sca_concat2.py Remove scalar+interval query from the “should error” list (behavior change).
test/cases/11-Functions/01-Scalar/test_fun_sca_concat.py Remove scalar+interval query from the “should error” list (behavior change).
test/cases/11-Functions/01-Scalar/test_fun_sca_concat_ws.py Remove scalar+interval query from the “should error” list (behavior change).
test/cases/11-Functions/01-Scalar/test_fun_sca_char_length.py Remove scalar+interval query from the “should error” list (behavior change).
test/cases/10-Operators/01-Arithmetic/test_arithmetic.py Convert previously-invalid interval arithmetic queries into validated successful queries.
test/cases/09-DataQuerying/14-Tags/test_tag_basic.py Convert some interval window tag projection queries from expected-error to expected-success.
test/cases/09-DataQuerying/10-Distinct/test_query_distinct.py Add positive coverage for DISTINCT ... INTERVAL(...) cases and remove them from the error list.
source/libs/planner/src/planSpliter.c Adjust stable split strategy for interval/external windows when running in projection-mode.
source/libs/planner/src/planLogicCreater.c Build window logic targets from projections for projection-mode; adjust handling for HAVING/slot binding.
source/libs/parser/src/parTranslater.c Detect window projection-mode and relax/adjust group-by/expr rewriting; enforce restricted FILL modes for projection-mode windows.
source/libs/parser/src/parTokenizer.c Add AGG and SCALAR to the SQL keyword table.
source/libs/parser/src/parAstCreater.c Add AST helper to set select-statement window mode.
source/libs/parser/inc/sql.y Extend SELECT grammar to accept optional SCALAR/AGG window-mode selector.
source/libs/parser/inc/parAst.h Declare setSelectStmtWindowMode.
source/libs/executor/src/timewindowoperator.c Pass projection list into indefinite-rows runtime initialization for window operators.
source/libs/executor/src/executorInt.c Extend indefinite-rows runtime to support a projection expression path (projSupp) and fill pseudo-columns for it.
source/libs/executor/src/eventwindowoperator.c Pass projection list into indefinite-rows runtime initialization for event windows.
source/libs/executor/src/countwindowoperator.c Pass projection list into indefinite-rows runtime initialization for count windows.
source/libs/executor/inc/executorInt.h Extend SIndefRowsRuntime and update initIndefRowsRuntime signature.
include/libs/nodes/querynodes.h Add EWindowMode, store windowMode, and add hasScalarExpr flag to SSelectStmt.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread source/libs/parser/inc/sql.y
Comment thread test/cases/query/in/test_window_projection_small.in Outdated
Comment thread test/cases/13-TimeSeriesExt/06-SessionWindow/test_session.py Outdated
Comment thread source/libs/planner/src/planLogicCreater.c Outdated
Copilot AI review requested due to automatic review settings May 9, 2026 09:00
@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from 8e60d32 to b8a6deb Compare May 9, 2026 09:00
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 40 out of 47 changed files in this pull request and generated 1 comment.

Comment thread source/libs/planner/src/planLogicCreater.c Outdated
@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from b8a6deb to 93f2a06 Compare May 11, 2026 07:13
Copilot AI review requested due to automatic review settings May 11, 2026 08:28
@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from 93f2a06 to 7232415 Compare May 11, 2026 08:28
@Simon9997 Simon9997 requested a review from zitsen as a code owner May 11, 2026 08:28
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 48 out of 55 changed files in this pull request and generated 3 comments.

Comment thread source/libs/parser/src/parTranslater.c
Comment thread source/libs/parser/src/parTranslater.c
Comment thread source/libs/executor/src/executorInt.c Outdated
@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from 7232415 to d293e3d Compare May 11, 2026 09:26
Copilot AI review requested due to automatic review settings May 12, 2026 00:38
@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from d293e3d to 76db787 Compare May 12, 2026 00:38
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 49 out of 56 changed files in this pull request and generated 2 comments.

Comment thread source/libs/planner/src/planSpliter.c Outdated
Comment thread test/cases/09-DataQuerying/10-Distinct/test_query_distinct.py
Copilot AI review requested due to automatic review settings May 12, 2026 02:13
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 49 out of 56 changed files in this pull request and generated 1 comment.

Comment thread source/libs/parser/src/parTranslater.c
@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from 8322898 to 7a31038 Compare May 12, 2026 02:47
Simon9997 added 3 commits May 12, 2026 13:40
- Relax window query SELECT list restrictions to allow raw columns
- Add SCALAR/AGG mode keywords to grammar
- Fix needFillImpl to exclude scan pseudo column functions (tbname)
  from fill value counting in partition-by queries
- Create projection expressions for window logical plans
- Handle fill value classification for projection mode
- Mark indefinite rows window in splitter for proper distribution
- Add projection support in indefinite rows window operator
- Sort window states by (groupId, skey) before closing to ensure
  correct fill order in partition-by queries
Simon9997 added 2 commits May 12, 2026 13:40
- Update SubQuery answer files for new column ordering
- Fix assertions in distinct, tag, arithmetic, scalar function tests
- Adjust state window and session window test expectations
- Add tests for default/SCALAR/AGG mode with fill(null/null_f/none/value/value_f)
- Add partition-by tests with tbname and tag columns
- Add edge case, large data, and pseudo-column-only fill tests
- Register test in CI cases.task
Copilot AI review requested due to automatic review settings May 12, 2026 05:40
@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from 7a31038 to aee3ff8 Compare May 12, 2026 05:40
@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from aee3ff8 to da59a30 Compare May 12, 2026 05:41
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 49 out of 56 changed files in this pull request and generated 4 comments.

Comment thread test/cases/09-DataQuerying/10-Distinct/test_query_distinct.py
Comment thread source/libs/parser/inc/sql.y
Comment thread source/libs/parser/src/parTranslater.c
Comment thread docs/en/05-basic/03-query.md Outdated
@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from da59a30 to b59c0ce Compare May 12, 2026 06:28
@dapan1121
Copy link
Copy Markdown
Contributor

Code Review — window projection mode (SCALAR/AGG keywords)

Reviewed by GitHub Copilot (tsdb-code-review skill v0.1.0)

Summary

This PR introduces window projection mode (v3.4.2.0): window queries may now include plain column expressions in the SELECT list, causing each window to output all original rows instead of one aggregated row. Two new SQL keywords SCALAR/AGG disambiguate mode when only pseudocolumns/tags/constants are selected.

5 verified issues found (2 BLOCKER, 3 MAJOR):


🔴 BLOCKER

1. O(N²) complexity — checkWindowProjectionMode called per node in tree walk
source/libs/parser/src/parTranslater.c line 5413

checkWindowProjectionMode is called inside doCheckExprForGroupBy (a nodesRewriteExprs tree-walk callback, invoked O(T) times), and itself runs nodesWalkExprs(pSelect->pProjectionList, …) — another O(T) walk. For a SELECT tree of total size T, the group-by semantic check phase now costs O(T²).

Fix: Compute isScalarMode once before the walk and pass it via STranslateContext, or cache it in SSelectStmt. Do not compute it inside a per-node callback.


2. SCALAR/AGG accepted silently on queries without a window clause
source/libs/parser/src/parAstCreater.c line 2770

There is no validation that windowMode != WINDOW_MODE_NONE requires a window clause to be present. SELECT SCALAR c1 FROM t1 WHERE ts > 0 parses and translates without error. If doCheckExprForGroupBy is subsequently invoked for that query (e.g., triggered by ORDER BY context), checkWindowProjectionMode returns true (WINDOW_MODE_SCALAR, no agg/groupby/indefrows, ctx.hasScalarExpr = true), causing all partition-key and tag-column rewriting to be silently skipped and producing an incorrect query plan.

Fix:

// In translateSelectFrom or a dedicated checkWindowMode():
if (pSelect->windowMode != WINDOW_MODE_NONE && pSelect->pWindow == NULL) {
    return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_OPTR_USAGE,
        "SCALAR/AGG can only be used with a window clause");
}

Also add || pSelect->pWindow == NULL to the early-return guard in checkWindowProjectionMode.


🟠 MAJOR

3. Potential memory leak of pProjExprInfo in initIndefRowsRuntime error path
source/libs/executor/src/executorInt.c line 1337–1340

createExprInfo allocates pProjExprInfo. If initExprSupp fails before it stores the pointer in pRuntime->projSupp.pExprInfo, the _return path's cleanupExprSupp(&pRuntime->projSupp) will not free it (because projSupp.pExprInfo is still NULL).

Fix: Transfer ownership immediately after allocation so cleanupExprSupp always handles it:

code = createExprInfo(pProjs, NULL, &pProjExprInfo, &numProjs);
QUERY_CHECK_CODE(code, lino, _return);
pRuntime->projSupp.pExprInfo = pProjExprInfo;  // transfer before initExprSupp
pProjExprInfo = NULL;
code = initExprSupp(&pRuntime->projSupp, pRuntime->projSupp.pExprInfo, numProjs, pFuncStore);
QUERY_CHECK_CODE(code, lino, _return);

Or verify that initExprSupp always stores the pointer before any failure point.


4. INTERVAL projection mode routes to stbSplSplitSessionOrStateForBatch — confirm correctness
source/libs/planner/src/planSpliter.c line ~935

In stbSplSplitWindowForCrossTable, INTERVAL windows in projection mode (!pWin->pFuncs && pWin->pProjs) are dispatched to stbSplSplitSessionOrStateForBatch, a function written for SESSION/STATE windows. These have different partitioning semantics from INTERVAL (no time-alignment, no cross-vnode merge). Using it for multi-vnode INTERVAL projection queries may produce incorrect data distribution or output ordering.

Fix: Add a multi-vnode integration test: SELECT _wstart, ts, c1 FROM stable PARTITION BY tbname INTERVAL(10s). If results are correct, add a comment explaining why this split strategy is intentionally reused here. If not, introduce a dedicated split path.


5. hasScalarExpr and checkWindowProjectionMode can diverge

There are two independent mode-detection mechanisms:

  • checkWindowProjectionMode re-walks pProjectionList looking for COLUMN_TYPE_COLUMN nodes
  • pSelect->hasScalarExpr is set in doCheckExprForGroupBy under !hasSelectFunc && !isRelatedToOtherExpr

These conditions are not equivalent and can diverge (e.g., a column that isRelatedToOtherExpr would be seen by checkWindowProjectionMode but not set hasScalarExpr). Downstream code in planLogicCreater.c gates pWindow->indefRowsFunc and projectionMode on hasScalarExpr, while the translator skip logic uses checkWindowProjectionMode. A mismatch between the two means the planner and the translator disagree on the query's mode.

Fix: Use a single cached field (e.g. bool windowScalarMode in SSelectStmt) as the sole source of truth, set once early in translateSelectFrom, and used by both the translator and the planner.


🟡 MINOR

  • line vs lino namingfillIndefRowsWindowPseudoCols uses line instead of lino (codebase convention, executorInt.c line 1297)
  • Removed negative test without commentparSelectTest.cpp line 337 drops run("SELECT c1 FROM t1 INTERVAL(10s)", TSDB_CODE_PAR_INVALID_OPTR_USAGE) without explanation; add a comment noting it is now valid in projection mode
  • Missing comment on FILL restriction scope — the FILL mode restriction check only handles INTERVAL_WINDOW; other window types silently pass without a note explaining that they cannot carry a FILL node syntactically

Comment thread docs/zh/14-reference/03-taos-sql/20-select.md Outdated
Comment thread source/libs/parser/src/parTranslater.c
Comment thread source/libs/parser/src/parAstCreater.c
Comment thread source/libs/executor/src/executorInt.c
Comment thread source/libs/planner/src/planSpliter.c
Comment thread source/libs/executor/src/executorInt.c
Comment thread source/libs/parser/test/parSelectTest.cpp
Copilot AI review requested due to automatic review settings May 13, 2026 06:30
@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from b59c0ce to 477bee7 Compare May 13, 2026 06:30
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 50 out of 57 changed files in this pull request and generated 1 comment.

@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from 477bee7 to dac016b Compare May 13, 2026 07:19
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 13, 2026 09:09
@Simon9997 Simon9997 force-pushed the feat/3.0/6659773700 branch from dac016b to 3b9e8ce Compare May 13, 2026 09:09
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 52 out of 59 changed files in this pull request and generated 2 comments.

Comment thread source/libs/planner/src/planLogicCreater.c
Comment thread source/libs/parser/src/parTranslater.c
@Simon9997 Simon9997 closed this May 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants